Imaging system having structural data enhancement for non-visible spectra

ABSTRACT

An imaging system includes a light source for emitting visible light and infrared light and a camera head unit configured to capture visible light image data so as to generate a visible light image frame and configured to capture infrared image data so as to generate an infrared image frame. A camera control unit is configured to extract a structural data from the visible light image frame. The camera control unit is further configured to apply the structural data to the infrared image frame so as to enhance the infrared image with structural data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/484,097 filed on Sep. 24, 2021, and entitled, “Infrared ImagingSystem Having Structural Data Enhancement,” which is in turn acontinuation of U.S. application Ser. No. 16/712,191 filed on Dec. 12,2019, titled “Infrared Imaging System Having Structural DataEnhancement,” and issued as U.S. Pat. No. 11,159,749 on Oct. 26, 2021;the entire contents all above named applications are incorporated hereinby reference.

TECHNICAL FIELD

The disclosure relates to an imaging system for medical procedures.

BACKGROUND

Surgical imaging systems may include one or more light sources that emitboth visible light and non-visible light such as infrared light. Thevisible light is typically used as a reference light or illuminatinglight, while the infrared light is typically used as an excitation lightthat excites a dye to emit light through fluorescence in the infraredspectrum. Visible light provides the surgeon with an image as seen bythe human eye, whereas the image generated by the fluorescence lightprovides an infrared image of tissue characteristics such as blood flow,inflammation and the like. A depiction of a visible light image (“VLimage”) and a non-visible light image, such as the infrared image (“IRimage”), is illustratively shown in FIGS. 1A and 1B, respectively.

It should be appreciated that the VL image provides a clearer view ofstructural features of a surgical site such as veins, different bodytissue and the like whereas the IR image provides a visual depiction ofaspects such as blood flow within the tissue and/or diseased tissue.FIG. 1B depicts how the IR image lacks the structural details shown inthe VL image. Thus, the two images are analyzed by the surgeon during amedical procedure. In some cases, the IR image is colored and thencombined with the VL image. For example, data processing hardware isconfigured to use a color map, such as a look-up table (LUT), togenerate a pseudo-color mapped image (colored IR image) based on the IRimage which is then combined with the VL image to create a combinedimage. This provides the surgeon the benefit of viewing the structuraldetails in context with tissue conditions such as blood flow or disease.However, the colored IR image obscures some of the structural detailprovided in the VL image when combined, as shown in the combined imageof FIG. 2 .

SUMMARY

Accordingly, it is desirable to enhance the IR image or colored IR imageto includes structural data so as to facilitate improved real timeanalysis of the surgical site.

One aspect of the disclosure provides an imaging system generating an IRimage enhanced with structural data. The system includes a light sourcefor emitting infrared light and visible light. A camera head unitcaptures VL image data so as to generate VL image frame and captures theIR image data so as to generate an IR image frame.

The imaging system also includes a camera control unit (“CCU”)configured to execute instructions for executing imaging functions. TheCCU is configured to extract a structural data from the VL image frame.The CCU is further configured to apply the structural data to the IRimage frames so as to enhance the infrared light video with theextracted structural data so as to generate an enhanced IR image frame.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the structuraldata is an edge identified in the VL image frame. The edge may beidentified using known image filters which detect a contrast in the VLimage frame. Such image filters illustratively include a bilateralfilter, a high-pass and/or a low-pass filter.

In some implementations, the CCU is further configured to detect thestructural data by converting the VL image frames into grayscale imageframes prior to extracting the structural data.

In some implementations, the CCU scales the structural data by apredetermined factor so as to generate an enhancement signal. Theenhancement signal is then applied to the IR image frame. Thus, theenhancement of the structural data shown in the enhanced IR image may beincreased or decreased based upon the enhancement signal.

In some implementations, the data processing hardware is configured togenerate a color map of the IR image data. The color map depicts variousintensities of a color based upon the diffusion of the dye within thetissue. The CCU is further configured to apply the structural data tothe color map.

In some implementations, the structural data is determined by processinga predetermined luminance identified in the VL image frame. This may bedone in conjunction with or in lieu of processing a contrast in the VLimage frame.

A method for generating an IR image enhanced with structural data isalso provided. The method may be implemented in an imaging system havingone or more light sources that emit both white (i.e., visible) light andinfrared light and a first image sensor and a second image sensor areconfigured to capture the visible light and the infrared lightrespectively.

The method includes the step of obtaining a VL image frame and obtainingan IR image frame. The method includes the steps of extracting astructural data from the VL image frame and applying the structural datato the IR image frame so as to generate an enhanced IR image frame.Accordingly, the enhanced IR image frame includes structural dataextracted from the VL image frame.

Implementations of the method disclosed herein may include one or moreof the following optional features. In some implementations, thestructural data is an edge identified in the VL image frame, wherein theedge may be identified by a contrast in the VL image. The contrast maybe determined using known filters, such as a bilateral filter, ahigh-pass and/or low-pass filter.

In some implementations, the method further includes the step ofconverting the VL image frames into grayscale image frames prior toextracting the structural data.

In some implementations, the structural data is scaled by apredetermined factor so as to generate an enhancement signal. Thus, theenhancement of the structural data shown in the enhanced IR image may beincreased or decreased based upon the enhancement signal.

In some implementations, the method includes the step of generating acolor map of the IR image data and applying the structural data to theIR image frame.

In some implementations, the structural data is determined by processinga predetermined luminance identified in the VL image frame.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the subject matter defined by theclaims. The patent or application file contains at least one drawingexecuted in color. Copies of this patent or patent applicationpublication with color drawing(s) will be provided by the Office uponrequest and payment of the necessary fee. The following description ofthe illustrative embodiments can be understood when read in conjunctionwith the following drawings, where like structure is indicated with likereference numerals and in which:

FIG. 1A is a view of a VL image frame shown in color.

FIG. 1B is a view of an IR image frame shown in color.

FIG. 2 a view of a combined image frame combining the VL image frameshown in FIG. 1A with a colored IR image frame based on the IR imageframe shown in FIG. 1B, shown in color.

FIG. 3 is a schematic view of an imaging system according to one or moreembodiments described herein.

FIG. 4 is a view of a structural data image frame showing the structuraldata taken from the VL image frame shown in FIG. 1A.

FIG. 5 is a view of an enhanced IR image frame based on the IR imageframe shown in FIG. 1B enhanced with the structural data image frameshown in FIG. 4 shown in color.

FIG. 6 is a schematic view of a CCU according to one or more embodimentsdescribed herein.

FIG. 7A is an illustrative view of structural data in a grayscale imageframe;

FIG. 7B is a view of the structural data further enhanced to increasedefinition of some structural data relative to FIG. 7A.

FIG. 7C is a view of the structural data further enhanced to decreasedefinition of some structural data relative to FIG. 7A.

FIG. 8A is a view of an IR image frame shown in color.

FIG. 8B is a view of a colored IR image frame based on the IR imageframe shown in FIG. 8B with a color map applied shown in color.

FIG. 8C is a view of an enhanced colored IR image frame based on thecolored IR image frame shown in FIG. 8B with structural data extractedfrom a VL image shown in color.

FIG. 9A is a view of a combined image frame based on a VL image framecombined with a colored IR image frame shown in color.

FIG. 9B is a view of an enhanced combined image frame based on thecombined image frame shown in FIG. 9A enhanced with structural dataapplied to the colored IR image frame shown in color.

FIG. 10 is a flow chart showing a method of generating an IR imageenhanced with structural data.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Implementations herein are directed toward an imaging system configuredto generate an IR image enhanced with structural data. In one aspect,the imagining system generates a VL image and extracts structural datafrom the VL image. The structural data is then applied to the IR imageso as to generate an enhanced IR image which provides clear visualcontext of anatomical features. As used herein, the term “applied”refers to the technique of overlaying one image frame with another,wherein anatomical features from one image frame are aligned withanatomical features of the other image frame.

For illustrative purposes, the imaging system is described herein withinthe context of endoscopy. Endoscopy, a nonsurgical medical procedureused to examine internal body cavities (e.g., a digestive tract), isincreasingly used as an effective diagnostic tool. The procedures aretypically performed using endoscopes, which include, in their most basicform, a flexible tube with a light source configured to emit visiblelight and infrared light. A CCU controls the exposure of a pair of imagesensors configured to detect light in the visible spectrum and infraredspectrum respectively. The flexible tube is passed through an orifice(e.g., the mouth) of a patient and the camera records images illuminatedby the light source.

FIG. 1A is an illustrative view of a VL image frame 100 taken from alaparoscopic procedure. The liver 200, bile duct 202 and the intestine204 are clearly distinguishable, as are the veins. For illustrativepurposes, an example of the imaging system will be made in the contextof a laparoscopic procedure.

In some procedures, non-visible imaging may also be used. For example,infrared light is used to provide fluorescent imaging. For instance,indocyanine green (“ICG”) is a cyanine dye used frequently in medicaldiagnostics and endoscopy for determining cardiac output, hepaticfunction, liver and gastric blood flow, and for ophthalmic angiography.For example, ICG distribution within the tissue enables intraoperativeevaluation of a tissue perfusion and vacuolization, identification ofcritical neurovascular structures and differentiation of tissue plainsbetween lesions and adjacent structures. ICG, when irradiated withinfrared light (750 nm and 950 nm), emits light through fluorescence.When viewed using an image sensor, the fluorescence generates an IRimage in the infrared spectrum. The IR image may be output by the sensoras a monochromatic IR image which obscures some of the structuraldefinition of the body, as shown in FIG. 1B when compared to FIG. 1A.

In FIG. 1B, the intensity of the image correlates to the diffusion ofthe dye. Thus, the intensity of the dye collected in the liver 200 andpassing through the bile duct 202 is more intense relative to theintestine 204. FIG. 1B also depicts how structural data such as theedges defining the liver 200 from the bile duct 202 and the intestines204 are not as clear as what is shown in FIG. 1A. Other structural datain the IR image, such as the veins 206 are also obscured by theirradiating dye. With reference now to FIG. 2 , the IR image may becolored using a color map to generate a colored IR image and thencombined with the visible light image shown in FIG. 1A to create acombined image. FIG. 2 illustrates how the color IR image obscures someof the details of the visible light image when combined. In particular,some of the edges of the liver 200, bile duct 202, and intestines 204near the colored IR image are shown as being generally blurry, anddetails such as the veins 206 are hidden.

With reference now to FIG. 3 , an imaging system 10, according to oneaspect, that generates an enhanced IR image frame enhanced withstructural data 12 (illustratively shown in FIG. 5 as depicted in agrayscale frame 36) is provided. For illustrative purposes, adescription of the imaging system 10 is provided within the context ofan endoscopic system. However, it should be appreciated that the imagingsystem 10 may be utilized in other applications, illustrativelyincluding an exoscope, borescope and other systems having two or moreillumination-types and one or more image sensors 14. Furthermore,although the imaging system 10 is described with respect to medicalapplications using fluorescing dye, it should be understood thatindustrial applications using other combinations of visible light andinfrared light of narrow wavelength ranges may benefit from the sameprinciples.

The imaging system 10 includes an endoscope 16, one or more lightsources 18, a camera head 20 and a CCU 22. In some examples, the lightsource 18 is included within the camera head 20. In other examples, theendoscope 16 and the camera head 20 may form an integral unit known inthe art as a videoscope. FIG. 3 provides an example where the camerahead 20 and the CCU 22 are formed as an integral unit, but it should beappreciated that the CCU 22 may be a separate unit in communication withthe camera head 20. Regardless of the specific configuration, theprinciples of the present disclosure apply to various examples of videoimaging systems 10 as previously noted. The camera head 20, light source18 and CCU 22 are coupled to each other wirelessly or through a wiredconnection.

The light source 18 emits both visible light (VL) and near infraredlight (NIR). In some examples, a timer 24 is provided to control theoperation of the light source 18 so as to switch between emitting VL andNIR light and the light source 18 is coupled to the endoscope 16 so asto illuminate the surgical site. The light source 18 may include one ormore light-emitting diodes (LEDs) or any other appropriatelight-emitting device. Separate light sources 18 may emit the VL and theNIR light respectively. In other examples, both VL and NIR may besimultaneously provided.

Light emitted by the light source 18 travels along a light guide (e.g.,an optical fiber) and, after exiting the light guide, illuminates orirradiates a target area of the surgical site (e.g., an internal cavityof a patient). Reflected VL (i.e., VL that has reflected from the targetarea) and emitted fluorescent light (i.e., light emitted by, forexample, ICG that has been irradiated by NIR light) are directed backthrough the optical pathway and are captured by the camera head 20 andprocessed as image data, VL image data and IR image data respectively.In particular, the image sensors 14 are controlled by the CCU 22 so asto be exposed during the emission of respective VL and IR light.

The image sensors 14 may be a complementary metal oxide semiconductor(CMOS) or a Charged Coupled Device (CCD). It should be appreciated thatany pixelated image sensor 14 currently known or later developed may bemodified and adopted for use herein. The image sensors 14, in someimplementations, include color filter arrays (CFAs).

The image sensors 14 transmit VL image data and IR image data to the CCU22. The CCU 22 is configured to execute instructions for performingimaging functions. For example, the CCU 22 may include computingresources 26 (e.g., data processing hardware 26 a, field programmablegate array (“FPGA”) and the like) for executing instructions configuredto perform image processing and storage resources 28 (e.g., memoryhardware 28 a). The CCU 22 processes the VL image data and the IR imagedata to produce a respective VL image frame 100 and an IR image frame102, as illustratively shown in FIG. 1A and FIG. 1B respectively. The IRimage frame 102 may be converted to a color mapped image frame (“coloredIR image frame”) as shown in FIG. 8B, wherein the colored IR image frameis then superimposed on the VL image as shown in FIG. 2 . In someimplementations, a display control unit 30 is further configured tocompile the VL image frames and the colored IR image frames so as togenerate respective VL and IR videos which may be displayed on a displayunit 32.

In some implementations, the CCU 22 is disposed physically within thecamera head 20 and in wired communication with the image sensors 14. Inother implementations, the CCU 22 is in wireless communication with theimage sensors 14 (e.g., via wireless, Bluetooth, etc.) and may be remotefrom the image sensors 14 and/or system. In this case, the CCU 22 maycorrespond to any appropriate computing device, such as a desktopworkstation, laptop workstation, or mobile device (e.g., smart phone ortablet). In yet other implementations, the data may be stored innonvolatile storage at the system (e.g., a thumb drive) and laterremoved to be processed at data processing hardware 26 a and storageresource 28 such as memory hardware 30 a remote from the image sensors14. The memory hardware 30 a stores instructions that when executed onthe data processing hardware 26 a cause the data processing hardware 26a to execute camera and imaging functions.

The CCU 22 processes (i.e., using instructions stored on the storageresources or written on an FPGA) the VL image frame 100 so as to extractstructural data 12 (illustratively shown in FIG. 4 in areas surroundedby dotted lines) and apply the structural data 12 to the IR image frame102 to create an enhanced IR image frame as shown in FIG. 5 . In someimplementations, the structural data 12 are edges identified in the VLimage frame. It should be appreciated that the VL and IR image frames100, 102 capture substantially the same target area, as shown in FIGS.1A and 1B.

FIG. 1A depicts a VL image frame, wherein the edges identified by theCCU 22 define the shape of the body tissue and clearly delineates thedifferent tissue and body parts from each other. For example, the edgesdefine the veins 206, liver 200, bile duct 202 and the intestine 204 ofthe target area.

With reference again to FIG. 4 , a depiction of the structural data 12taken from the VL image frame 100 shown in FIG. 1A is provided. The CCU22 executes instructions, vis-a-vis a data processing hardware 26 a oran FPGA, to extract the structural data 12 from the VL image frame 100.FIG. 4 illustratively shows lines defining the edges of the liver 200,bile duct 202 and the intestine 204, as well as the veins 206.

In one aspect, the edges may be identified using known image filters 34which may be written as a set of instructions stored in the memoryhardware 30 a or FPGA. The image filters 34 may be formed as part of theCCU 22. The image filters 34 may be configured to detect a contrastlevel in the VL image frame 100. Such image filters 34 illustrativelyinclude a bilateral filter, a high-pass and/or a low-pass filter whichprocesses the VL image frame 100 so as to produce the structural data 12illustratively shown in FIG. 4 .

Referring now to FIG. 6 , in one aspect, the CCU 22 converts the VLimage frame 100 into a grayscale image frame 36 as depicted in FIG. 4 .Any such an image processing function currently known or later developedto convert a VL image frame 100 into grayscale image frame 36 may beadapted for use herein. For use herein, the term “grayscale image frame”means an image frame which has been processed such that the value ofeach pixel is a single sample representing only an amount of light, thatis, it carries only intensity information so as to only include shadesof gray. Thus, the colors (red, green and blue for instance) are removedfrom the VL image frame 100 and the image is depicted in terms of theintensity of gray. The CCU 22 extracts the structural data 12 from thegrayscale image frame 36 and applies the structural data 12 to the IRimage frame (such as the one shown in FIG. 1B), so as to generate anenhanced IR image frame enhanced with structural detail as shown in FIG.5 . One benefit of extracting the structural data 12 from a grayscaleimage frame 36 is that the edges are more easily detected. This isclearly shown when comparing FIG. 5 with FIG. 1B. In particular, theveins, liver, bile duct and intestine shown in FIG. 5 are more distinctrelative to the same part shown in FIG. 1B.

FIG. 6 provides a schematic view of the CCU 22 showing the process forgenerating an IR image frame enhanced with structural details. The VLimage sensors 14 a and the IR image sensors 14 b are shown as generatinga respective VL image frame 100 and an IR image frame 102, such as VLand IR image frames 100, 102 illustratively shown in FIGS. 1A and 1Brespectively. The VL image frame 100 is generated using a color imagefilter 38, such a Bayer filter to generate a VL image frame 100 havecolors in three color channels-Red, Green and Blue (“RGB”). The CCU 22converts the VL image frame 100 into a grayscale image frame 36. Anedge-preserving low pass filter 34, such as a bilateral filter,separates the structural data 12 from the grayscale image frame 36,which may be executed by a FPGA so as to extract the structural data 12,illustratively shown in FIG. 4 . As described above, the CCU 22 appliesthe structural data 12 onto the IR image frame, as shown in FIG. 5 .

In some implementations, the grayscale image frame 36 is generated byprocessing a luminance information of the VL image frame 100. Theluminance information may be taken from an average of the three colorchannels (“RGB”) of the VL image frame 100, or a single color, e.g. red,blue or green. The luminance information as is known in the art is anintensity of the VL image data for each pixel. In such a manner, it ispossible for determining structural data 12 using luminance information.

As stated above, the grayscale image frame 36 may be generated by asingle color, wherein the VL image frame 100 is processed such that thevalue of each pixel is a single sample representing only an amount oflight. That is, the value of each pixel carries only intensityinformation so as to only include a shade of a single color, such asred, green or blue. The CCU 22 processes the image data from the singlecolored image frame so as to generate a grayscale image frame 36. Itshould be appreciated that any process known or later developed togenerate a grayscale image frame 36 may be adapted for use herein.

With reference again to now to FIGS. 7A-7C, another aspect of theimaging system 10 is provided. In such an aspect, the CCU 22 adjusts theenhancement of the structural data 12 by a predetermined factor byapplying an enhancement signal. FIG. 7A shows the grayscale image frame36 wherein the structural data 12 is not further enhanced. FIG. 7B showsan example where enhancement of the structural data 12 is increased oremphasized, for example by increasing the edge widths or intensities bya factor greater than 1, and thus the edges are sharper and darkerrelative to the edges shown in FIG. 7A. FIG. 7C shows an example whereenhancement of the structural data 12 is decreased or deemphasized, forexample by decreasing the edge widths or intensities by a factor lesserthan one (1) but greater than zero (0), and thus the edges are thinnerand dimmer relative to the edges shown in FIGS. 7A and 7B. It should beappreciated that the enhancement of the structural data 12 is depictedby an increase or decrease in the thickness for illustrative purposes,and that the scaled structural data 12 may be displayed in terms of anincrease or decrease in intensity.

The predetermined factor may be adjusted using an input 40 such as adial, a keyboard, touchscreen voice command or the like. Thus, theenhancement of the structural data 12 shown in the enhanced IR image ofFIG. 5 may be increased or decreased based upon the enhancement signal.The input may be actuated in real time, adjusting the predeterminedfactor so as to adjust the enhancement signal in real time. As such, theenhancement of the structural data 12 may be adjusted based upon theintensity of the fluorescence in the IR image frame. For instance,should the intensity of the fluorescence in the IR image frame obscurethe structural data 12, the enhancement signal may be increased by theinput so as to better delineate the different tissues within thesurgical site, e.g. distinguish the liver from the bile duct and theintestines.

With reference now to FIGS. 8A-8C, an aspect of the imaging system 10 isdescribed wherein the IR image frame is colored. In such an aspect, theCCU 22 is configured to generate colored IR image frame using a colormap applied to the IR image data, the color map having a plurality ofcolor planes, any such an image processing function currently known orlater developed to generate a colored IR frame may be adapted for useherein. FIG. 8A shows the surgical site of an IR image frame asprocessed by the CCU 22. FIG. 8B shows the colored IR image frame havinga color map applied which is generated by the CCU 22. For illustrativepurposes, the color map is shown as being various shades of green;however, it should be appreciated that other colors may be usedincluding multiple colors. FIG. 8B illustrates how the colored IR imagefurther obscures the details of the surgical site, relative to the IRimage shown in FIG. 8A. Thus, the delineation of the bile duct 202 andthe liver 200, for example is unclear. The CCU 22 is configured to applythe structural data 12 to the colored IR image, as shown in FIG. 8C.FIG. 8C shows the edges of the liver 200, bile duct 202 and intestines204 are more distinct relative to FIGS. 8A and 8B. FIG. 8C further showsthe vein 206, which are obscured in FIGS. 8A and 8B.

With reference now to FIGS. 9A and 9B another aspect of the imagingsystem 10 is provided, wherein a combined image frame is generated byapplying the colored IR image onto the VL image. In particular, coloredIR image frame 102 has been processed by application of a color map asdescribed above. The structural data 12 is extracted from the VL imageframe 100 as described above. For instance, the CCU 22 may process theVL image frame 100 so as to generate a grayscale image frame 36 as shownin FIG. 4 , wherein the structural data 12 is extracted and then appliedto the colored IR image to generate an enhanced colored IR image frame.The enhanced colored IR image frame is then combined with the VL imageframe 100 so as to form an enhanced combined image frame that isenhanced with structural details as shown in FIG. 9B. Preferably, theCCU 22 performs cropping or filtering features to remove any structuraldata 12 from being applied to the VL image portion of the overlaid imageframe. A comparison of FIGS. 9B and 9A shows the advantage of applyingthe structural data 12 to the color map. In particular, FIG. 9Aillustrates how the edges delineating the liver 200 from the bile duct202 and the intestine 204 are obscured by the color map. However, thestructural data 12 of the liver 200, bile duct 202 and the intestines204 are more visible in FIG. 9B relative to FIG. 9A as the edgesextracted from the VL image frame 100 are applied to the colored IRframe portions of the enhanced combined image frame.

The structural data 12 may be extracted from the VL image frames 100using any of the techniques described above, that is the structural data12 may be an edge detected by filters, conversion of the VL image frames100 into grayscale, and luminance information by processing an averageof all the colors or a single colored VL image frame 100. The extractedstructural data 12 may be adjusted by a predetermined factor which maybe adjusted by an input so as to increase, decrease or remove thestructural data 12 applied to the colored IR image frames 102,illustratively shown in FIG. 8C.

In another aspect, the camera head 20 is configured to capture VL imagedata so as to generate at least two VL image frames 100 and configuredto capture IR image data so as to generate at least two IR image frames102. The CCU 22 is configured to compile the VL image frames so as toproduce a visible light video and compile the colored IR image frames soas to produce colored IR video. The CCU 22 is further configured toapply the structural data 12 to the IR image frames so as to enhance thecolored IR video. The CCU 22 may combine VL light frames 100 andenhanced colored IR frames to generate enhanced combined video. Thus,the imaging system 10 provides a video of the surgical site in both VLand infrared light, wherein frames based on the infrared light (whetherIR image frames or colored IR image frames) are enhanced with structuraldata 12 taken from the VL image frames.

With reference now to FIG. 10 , a method 300 for generating an IR imageframe enhanced with structural data 12 is also provided. The method 300may be implemented in the imaging system 10 described herein, whereinthe imaging system 10 includes one or more light sources 18 that emitboth white (i.e., visible) light and infrared light. A camera head 20having a first image sensor 14 a and a second image sensor 14 b areconfigured to capture the visible light and the infrared lightrespectively.

The method 300 includes the steps 302 a, 302 b of obtaining a VL imageframe 100 and obtaining an IR image frame 102. The VL image frame 100and the IR image frame 102 may be generated by the CCU 22. The method300 includes the step 304 of detecting and extracting structural data 12from the VL image frame 100. The method 300 includes the step 306 ofapplying the structural data 12 onto the IR image frame 102 by the CCU22 to generate an enhanced IR image frame. As described above, this maybe done using a FPGA and/or a data processing hardware 26 a.Accordingly, the enhanced IR image frame 102 includes structural data 12extracted from the VL image frame 100.

The structural data 12 may be an edge identified in VL image frame 100,wherein the edge may be identified by detecting a contrast in the VLimage, as shown in step 304 a. The contrast may be detected using knownfilters, such as a bilateral filter, high-pass or low-pass filter. Theextracted structural data 12 is converted into a structural data imageframe 12 a.

In some implementations, the step of extracting the structural data 12includes converting the VL image frames 100 into grayscale image frames36 prior to extracting the structural data 12. In some implementations,the step of extracting the structural data 12 includes obtainingluminance information from the VL image frame 100, as shown in step 304b. The luminance information may be taken from an average of the threecolor channels (“RGB”) of the VL image frame 100. Alternatively, theluminance information is taken from a single color. In such an aspect,the VL image frame 100 is processed to remove all but one color.

In some implementations, the structural data 12 is scaled by apredetermined factor so as to generate an enhancement signal, as shownin step 308. Thus, the enhancement of the structural data 12 shown inthe enhanced IR image frame 102 may be increased or decreased based uponthe enhancement signal. As described above, the predetermined factor maybe adjusted manually so as to increase or decrease the visibility of thestructural data 12 on the IR image frame 102.

In some implementations, the method 300 includes the step of generatinga color map of the IR image data, the color map having a plurality ofcolor planes and the data processing hardware 26 a is configured toapply the structural data 12 to the IR image frame 102, as shown in FIG.8C.

In some implementations, the method 300 includes the step of generatinga combined image frame having the colored IR image frame combined withthe VL image. In such a step, the colored IR image frame has beenprocessed by application of a color map to the IR image frame. Thestructural data 12 is extracted from the VL image frame 100 and appliedonto the colored IR image frame and combined with the VL image frame 100so as to generate the enhanced combined image frame.

The method 300 may include the step of cropping or filtering features toremove or prevent any structural data 12 from being applied to VL imageportions of the combined image frame. Such a step may help the VL imageretain a depiction of the surgical site as seen by the human eye.

The method 300 may further include the step of obtaining at least two VLimage frames 100 and at least two IR image frames 102. The CCU 22 isconfigured to compile the VL image frames 100 so as to produce a visiblelight video and compile the IR image frames 102 so as to produce aninfrared light video, the structural data 12, the CCU 22 is furtherconfigured to apply the structural data 12 to the IR image frames so asto enhance the colored light video. Thus, the method 300 provides avideo image of the surgical site in both VL and infrared light, whereinthe infrared light is enhanced with structural data 12 taken from the VLimage frames 100.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

1. An imaging system configured to display an image onto a display unit,the imaging system comprising: a light source for emitting a visiblelight and a non-visible excitation light; a camera head unit configuredto capture visible light image data so as to generate a visible lightimage frame and configured to capture an emission non-visible image dataso as to generate a non-visible emission image frame; and a cameracontrol unit configured to extract a structural data from the visiblelight image frame and apply the structural data to the non-visibleemission image frame so as to enhance the non-visible image frame withthe structural data.
 2. The imaging system of claim 1, wherein thenon-visible light is in the near infrared spectrum.
 3. The imagingsystem of claim 1, wherein the camera control unit is configured tocombine the visible light frame with the non-visible emission imageframe.
 4. The imaging system of claim 1 further comprising a display,displaying the visible light image frame and the non-visible imageframe.